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Abstract 

This paper reports on the development to date of the numeracy conceptual framework for the Adult Literacy 
and Lifeskills (ALL) Project. The inclusion of a numeracy scale in the ALL survey offered a significant 
opportunity to develop a new conceptual framework for adult numeracy. This paper discusses the meaning of 
numeracy and the notion of numerate behavior , presents a framework of five facets , and looks in some detail at 
a complexity-rating scheme used to guide construction of the assessment tasks that looks at the question of what 
makes one numeracy task more difficult than another. 

The ALL Project 

An international survey of the numeracy abilities of adults is to be part of the Adult Literacy and Lifeskills 
(ALL) Project, formerly called the International Life Skills Survey (ILSS), planned for the year 2002. This 
comparative survey is being jointly developed by Statistics Canada and by the United States’ National Center 
for Education Statistics (NCES), in cooperation with the Organization for Economic Cooperation and 
Development (OECD). The ALL project is a follow-up to the International Adult Literacy Survey (IALS), the 
world’s first large-scale comparative assessment of adult literacy. 

<f 

Tasks will assess performance in several skill domains, including Numeracy as well as Document and Prose 
Literacy, and Problem Solving, while other variables will be assessed via a background questionnaire. 

To this date, items and background questions developed for the survey have undergone a feasibility study in the 
Netherlands and the U.S. After translation of the materials, a pilot study will be administered in the 
participating countries in 2001 . The first round of the actual survey is scheduled for 2002. 

Why Include Numeracy? 

Numeracy is becoming a growing concern for diverse sectors. Countries are increasingly attending to topics 
such as improving workplace efficiency and quality processes, to resulting lifelong learning needs, and to civic 
participation. It is seen as vital that nations have information about their citizens’ numeracy, among other skills, 
if they want to plan effective education and lifelong learning opportunities. 

The concept of numeracy is also specifically related to the dialogue about the goals and especially outcomes 
and impact of school mathematics education. More educators now encourage links between knowledge gained 
in the mathematics classroom and students’ ability to handle real-life situations that require mathematical or 
statistical knowledge and skills. However, while numeracy may be a key skill area, its conceptual boundaries, 
cognitive underpinnings, and assessment have not received much prior scholarly attention. 
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What Is Adult Numeracy? 

One of the scales of the International Adult Literacy Survey (IALS), the Quantitative Literacy Scale, was a 
measurement of the respondent’s ability to apply arithmetic operations to numbers embedded in diverse texts. While 
this scale produced useful data, survey developers recognized that it was limited in scope. The Numeracy scale of 
ALL is designed to go above and beyond the QL Scale, while avoiding reliance on formal, curriculum-based 
knowledge of mathematics. 

IALS, following on a framework established in previous studies in the U.S. and Canada, made use of three 
literacy scales, Prose Literacy, Document Literacy, and Quantitative Literacy, to make operational its 
conception of literacy. The general definition of Literacy used in IALS was: 

Using printed and written information to function in society, to achieve one’s goals, and to 
develop one’s knowledge and potential. 

The definition for Document Literacy (DL) was: 

The knowledge and skills required to locate and use information contained in various formats, 
including job applications, payroll forms, transportation schedules, maps, tables, and graphics. 

The definition for Quantitative Literacy (QL) in IALS was: 

The knowledge and skills required to apply arithmetic operations, either alone or sequentially, 
to numbers embedded in printed materials. 

The working definition we have for numeracy in the ALL is: 

The knowledge and skills required to effectively manage the mathematical demands of 
diverse situations. 

Facets of Numeracy 

We have sought a view of numeracy that acknowledges the diverse purposes served by adults’ mathematical 
(and statistical) knowledge, that encompasses the different suggestions regarding the skills adults need to 
effectively function in home, work, community, and other contexts, and that takes into account the cognitive, 
metacognitive, and dispositional processes that support or affect adults’ numeracy. 

Overall, numeracy is a multifaceted and sometimes slippery construct. Our basic premise is that numeracy is the 
bridge that links mathematical knowledge, whether acquired via formal or informal learning, with functional 
and information-processing demands encountered in the real world. An evaluation of a person’s numeracy is far 
from being a trivial matter, as it has to take into account task and situational demands, type of mathematical 
information available, the way in which that information is represented, prior practices, individual dispositions, 
cultural norms, and more. 

A full assessment of all elements of such a broad conception is beyond the scope of ALL. We have thus chosen 
to focus on numerate behavior, which is revealed in the response to mathematical information that may be 
represented in a range of ways and forms. The nature of a person’s responses to mathematical situations 
depends on the activation of various enabling knowledge bases, practices, and processes. 

Table 1 presents our elaboration of numerate behavior. It has been used to guide development of items for a 
Numeracy Scale for the ALL. The statement in Table 1 distinguishes five facets, each with several components. 
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Table 1: Numerate Behavior and Its Facets 



Numerate behavior involves: 

Managing a situation or solving a problem in a real context 

everyday life 
work 
societal 

further learning 

by responding 
identifying or locating 
acting upon 

• order/sort 

• count 

• estimate 

• compute 

• measure 

• model 
interpreting 

communicating about 

to information about mathematical ideas 
quantity & number 
dimension & shape 
pattern and relationships 
data & chance 
change 

that is represented in a range of ways 
objects & pictures 
numbers & symbols 
formulae 
diagrams & maps 
graphs 
tables 
texts 

and requires activation of a range of 
enabling knowledge, behaviors, and processes 
mathematical knowledge and understanding 
mathematical problem-solving skills 
literacy skills 
beliefs and attitudes. 



A comparison of these facets of numeracy with what was included in the previous concept and definition of 
quantitative literacy of the IALS demonstrates how far the ALL concept of numeracy goes above and beyond the 
QL Scale. While the IALS QL scale depended on stimuli that were text-based and often required considerable literacy 
skills, the numeracy scale includes stimuli that span the range from minimum text dependency to high. The IALS QL 
scale only assessed the application of arithmetic operations while the numeracy scale includes the assessment of a 
broad range of skills and knowledge. 
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Complexity Factors 

A scheme of five factors was developed to account for the difficulty of different tasks, enabling an explanation 
of observed performance in terms of underlying cognitive factors. 

These five factors are: (1) Type of match/problem transparency; (2) Plausibility of distractors (including in 
text); (3) Complexity of mathematical information/data; (4) Type of operation/skill; (5) Expected number of 
operations. These factors have been used to attempt to estimate, separately and in interaction, the difficulty 
level of the numeracy tasks. Each numeracy task or item that was developed for the test was given a rating 
between 5 and 19, the sum of the numbers assigned for each of the factors as outlined in the flow chart. 

Diagram 1: Complexity Flow Chart 



Factor 




Each of these five factors has been developed in detail in the following tables. 
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Scoring for Each of the Complexity Factors 



Complexity Factor 1. Type of match/Problem transparency 
How difficult is it to identify and decide what action to take? How many literacy skills are required? 


Score 1 


Score 2 


Score 3 


In the question and the stimulus, the 
information, activity or operation 
required: 

- is clearly apparent and explicit — 
and all required information is 
provided 

- is specified in little or no text, 
using familiar objects and/or 
photographs or other clear, 
simple visualizations 

- is about locating obvious 
information or relationships only 

- closed question — not open-ended 


In the question and the stimulus, 
the information, activity or 
operation required: 

- is given using clear, simple 
sentences and/or visualizations 
where some translation or 
interpretation is required 

- is located within a number of 
sources within the text/activity. 

- fairly closed question 


In the question and the stimulus, the 

information, activity or operation 

required: 

- is embedded in text where 
considerable translation or 
interpretation is required 

- and/or 

- may need to be derived or 
estimated from a number of 
sources within or outside the 
text/activity 

- and/or 

- the information or action 
required is not explicit or 
specified 

- more complex, open-ended task 



Complexity Factor 2. Plausibility of distr 
How many other pieces of mathematical information are present? Is al 


actors 

! the necessary information there? 


Score 1 


Score 2 


Score 3 


- no other mathematical 
information is present apart from 
that requested — no distractors 


- there is some other 
mathematical information in the 
task that could be a distractor 

- the mathematical information 
given or requested can occur in 
more than one place 

- may need to bring to the 
problem simple information or 
knowledge from outside the 
problem. 


- other irrelevant mathematical 
information appears 

- mathematical information given 
or requested appears in several 
places. 

- necessary information or 
knowledge is missing, so outside 
information or knowledge needs to 
be brought in 



Complexity Factor 5. Expected number of operations 
How many steps and types of steps are required? 


Score 1 


Score 2 


Score 3 


one operation, action or process 


application of two or three steps, 
the same or similar operation, 
action or process 


integration of several steps covering 
more than one different operation, 
action or process 
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Complexity Factor 3. Complexity of mathematical informatioo/aoswer required 
How complex is the mathematical information that needs to be manipulated? 
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How complex is the mathematical action that is required? 
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The development of a scale that attempts to predict the complexity/difficulty of numeracy tasks was one of the 
more exciting, and challenging, aspects of the project. 

Results and Next Steps 

Based on the conceptual framework, over 100 items were developed and tested in a feasibility study in the U.S. 
and the Netherlands. A pool of 80 items remains that includes tasks at diverse levels of difficulty and that 
covers key facets of the conceptual framework for numeracy. For those 80 items, we found that the theoretical 
factors that were predicted to account for task difficulty were strongly correlated with the observed difficulty of 
items. A scatter plot of the data shows a significant negative linear correlation with an r of -0.859. Thus, about 
74% of the variation in performance on the 80 items is accounted for by the complexity factors. 

Preliminary results therefore provide initial support for the content validity and the construct validity of the 
numeracy scale. More reliable data are expected from the pilot study and the main survey. 

The next steps are to revise the complexity scheme in the light of feedback received to date, and to ask other 
interested teachers or researchers to rate items using the complexity scheme. Initially this could be to test the 
inter- rater reliability of the scheme on the 80 pilot study items. A further validation stage could be to test the 
scheme on new sets of items, and compare the theoretical predictions of difficulty levels with the actual 
performance of the items with sample respondents. 
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